Evalutating Students’ Code as a Learning Product

Today’s layout


  1. How has students’ code been analyzed?
  1. Another (qualitative) way to analyze students’ code
  1. Examples of how this framework could be used
http://bit.ly/imsi-coding-code

A bit about me…

“Supporting Data-Intensive Environmental Science Research: Data Science Skills for Scientific Practitioners of Statistics”

An image of an oceanside landscape, with a field of blooming California poppy flowers in the background.

How has students’ code been analyzed?

In the beginning there was…

An silouette of a blue jay.

Then came

A comparison of formula and tidyverse syntaxes (McNamara 2023)

  • lines of code produced
  • unique functions used

And finally

Rafalski et al. (2019) extended these same ideas to compare students’ ability to write accurate code across three different R syntaxes: the tidyverse, base R, and the tilde style.

  • number of errors
  • time completion

An alternative way to analyze students’ code

The Importance of Students’ Attention to Program State (Lewis 2012)

  • Attends to both the code produced by a student and their learning process

  • Pairs a student’s code with their debugging behavior side-by-side

A student looking sad while typing on a computer, the image of a bug over the top of their code is previewed above their head.

These analyses of students’ code should not be few and far between. Students’ code poses a unique avenue for qualitative research in the teaching and learning of computing.

Warm-up (90 seconds)

What process would you expect a student to use to create a multivariate scatterplot, with different colors for different groups?

A framework for analyzing student’s code (Schulte 2008)

Text Surface Program Execution Function
Macrostructure Understanding the overall structure of the program Understanding the “algorithm” of the program Understanding the goal / purpose of the program (in its context)
Relations References between blocks, e.g., method calls, object creation Sequence of method calls, object sequence diagrams Understanding how sub-goals are related to goals, how function is achieved by subfunctions
Blocks Regions of interest (ROI) that syntactically or semantically build a unit Operation of a block, a method, or a ROI (as a sequence of statements) Function of a block, may be seen as a sub-goal
Atoms Language elements Operation of a statement Function of a statement, only understandable in context

Coding student’s code


RPMA2GrowthSub$Weight[RPMA2GrowthSub$Age == 1]


Descriptive code

“Filters a vector of values using extraction operator, based on an equality relation with a variable selected from dataframe using $ operator”

In-vivo code

“Uses [ ] and == to filter vector, uses $ to select variable”

Uncovering emergent themes

linearAnterior <- lm(PADataNoOutlier$Lipid ~ PADataNoOutlier$PSUA)

early <- subset(RPMA2Growth, StockYear < 2006)  

Weight5 <- mean(RPMA2GrowthSub$Weight[RPMA2GrowthSub$Age == 5], na.rm = TRUE)

gas <- gas[!(substr(gas$sampleID,3,3) %in% c("b","c")), ]   

obsD <- subset(gas, gas$carboy == "D")$N15_N2_Ar

lowerCIBound <- pMat[1:mlleIndex,1][which.min(abs(mlleCI+likelihoods[1:mlleIndex]))]

Data wrangling

Statements of code whose purpose is to prepare a dataset for analysis and / or visualization

Sub-themes

  • selecting variables
  • filtering observations
  • mutating variables

An alternative direction


Process coding:

uses gerunds (“-ing” words) to connote action in the data (Saldana 2013)


  • Particularly relevant to describing the processes of human actions
  • Can be intertwined with time, such that actions can emerge, change, or occur in particular sequences.

Practical considerations

How much code should I collect?

  • Driven by the research question!
    • Amount of each student’s code
    • Number of students

How do readers trust my analysis?

  • Trust comes from:

    • confirmability
    • reliability
    • credibility
    • transferability


Excellent resources: Creswell & Poth (2018); Merriam & Tisdell (2016); Miles et al. (2020)

How could this be used?

Concept dependence

How does a student’s concept model of a dataset inform how they filter data?

(atoms; program execution)

Program environment

How do the visualizations produced by students who learn ggplot differ from those who learn “base” R?

(blocks; program execution)

Linguistic structure

How do students name objects they will use later?

(relationships; text)

Learning trajectory

How do students’ exploratory data analyses change over the duration of a course?

(macrostructure; function / purpose)

Why is this important for data science education?

Theobold et al. (2023)


How can we distinguish merely interesting learning from effective learning (Wiggins and McTighe 2005)?

Questions?

References

Corbin, Joseph, and Allan Strauss. 2008. Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks: Sage.
Creswell, J. W., and C. N. Poth. 2018. Qualitative Inquiry & Research Design. Thousand Oaks, CA: Sage.
Jadud, M. C. 2005. “A First Look at Novice Compilation Behavior Using BlueJ.” Computer Science Education 15 (1): 25–40. https://doi.org/10.1080/08993400500056530.
———. 2006. “Methods and Tools for Exploring Novice Compilation Behavior.” In Proceedings of the 2nd International Workshop on Computing Education Research (ICER), 73–84. Canterbury, United Kingdom: Association of Computing Machinery. https://doi.org/10.1145/1151588.1151600.
Lewis, Colleen M. 2012. “The Importance of Students’ Attention to Program State.” Proceedings of the Ninth Annual International Conference on International Computing Education Research, September. https://doi.org/10.1145/2361276.2361301.
McCall, D., and M. Kolling. 2014. “Meaningful Categorisation of Novice Programer Errors.” In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, 1–8. Madrid, Spain. https://doi.org/10.1109/FIE.2014.7044420.
McNamara, A. 2023. “Teaching Modeling in Introductory Statistics: A Comparison of Formula and Tidyverse Syntaxes.” ArXiv.
Merriam, S. B., and E. J. Tisdell. 2016. Qualitative Research. San Francisco, CA: John Wiley & Sons.
Miles, M. B., A. M. Huberman, and J. Saldaña. 2020. Qualitative Data Analysis. Thousand Oaks, CA: Sage.
Rafalski, T., P. M. Uesbeck, C. Panks-Meloney, P. Daleiden, W. Allee, A. McNamara, and A. Stefik. 2019. “A Randomized Controlled Trial on the Wild Wild West of Scientific Computing with Student Learners.” In Proceedings of the 2019 ACM Conference on International Computing Education Research, 239–47. https://doi.org/10.1145/3291279.3339421.
Saldana, J. 2013. The Coding Maual for Qualitative Researchers. Thousand Oaks: Sage.
Schulte, Carsten. 2008. “Block Model.” Proceedings of the Fourth International Workshop on Computing Education Research, September. https://doi.org/10.1145/1404520.1404535.
Theobold, Allison S., Megan M. Wickstrom, and Stacey A. Hancock. 2023. “Coding Code: Qualitative Methods for Investigating Data Science Skills.”
Wiggins, G., and J. McTighe. 2005. Understanding by Design. 2nd ed. Alexandria: Association for Supervision; Curriculum Development (ASCD).